Imaging system, driving assistance system, and program

ABSTRACT

The driving assistance system includes an imaging device capable of capturing a first monochrome image in a vehicle traveling direction, a first neural network for segmentation processing, a second neural network for depth estimation processing, a determination portion determining a center of a portion cut off from the first monochrome image on the basis of the segmentation processing and the depth estimation processing, a third neural network for colorization processing of only a second cut-off monochrome image, and a display device for enlargement of the second monochrome image subjected to the colorization processing.

TECHNICAL FIELD

One embodiment of the present invention relates to a neural network andan imaging system using the neural network. Another embodiment of thepresent invention relates to an electronic device using a neuralnetwork. Another embodiment of the present invention relates to avehicle using a neural network. Another embodiment of the presentinvention relates to an imaging system that obtains a color image byusing an image processing technique from a monochrome image obtained ina solid-state imaging element. Another embodiment of the presentinvention relates to a video monitoring system or a security systemusing the imaging system, a safety information service system, or adriving assistance system.

Note that one embodiment of the present invention is not limited to theabove technical field. One embodiment of the invention disclosed in thisspecification and the like relates to an object, a method, or amanufacturing method. One embodiment of the present invention relates toa process, a machine, manufacture, or a composition of matter.Therefore, specific examples of the technical field of one embodiment ofthe present invention disclosed in this specification and the likeinclude a semiconductor device, a display device, a light-emittingdevice, a power storage device, a storage device, an electronic device,a lighting device, an input device, an input/output device, a drivingmethod thereof, and a manufacturing method thereof. One embodiment ofthe present invention relates to a vehicle or an electronic device forvehicles provided in a vehicle.

Note that in this specification and the like, a semiconductor devicerefers to any device that can function by utilizing semiconductorcharacteristics. A transistor and a semiconductor circuit areembodiments of semiconductor devices. In addition, in some cases, astorage device, a display device, an imaging device, or an electronicdevice includes a semiconductor device.

Another embodiment of the present invention relates to a program using aneural network.

Another embodiment of the present invention relates to a driving systemin which a vehicle such as a motor vehicle can freely switch between asafety-assisted driving state, a semi-autonomous driving state, and anautonomous driving state.

BACKGROUND ART

A technique for forming a transistor by using an oxide semiconductorthin film formed over a substrate has attracted attention. For example,an imaging device with a structure in which a transistor that includesan oxide semiconductor and has an extremely low off-state current isused in a pixel circuit is disclosed in Patent Document 1.

In addition, a technique for adding an arithmetic function to an imagingdevice is disclosed in Patent Document 2.

REFERENCE Patent Documents

-   [Patent Document 1] Japanese Published Patent Application No.    2011-119711-   [Patent Document 2] Japanese Published Patent Application No.    2016-123087

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

With technological development, a high-quality image can be easilycaptured by an imaging device provided with a solid-state imagingelement such as a CMOS image sensor. In the next generation, an imagingdevice is required to be equipped with more intelligent functions.

An object of one embodiment of the present invention is to provide animaging device capable of image processing. Another object is to providean imaging device capable of high-speed operation. Another object is toprovide an imaging device with low power consumption. Another object isto provide a highly reliable imaging device. Another object is toprovide a novel imaging device or the like. Another object is to providea method for driving the imaging device. Another object is to provide anovel semiconductor device or the like.

Another object of one embodiment of the present invention is to providea driving assistance system suitable for a semi-autonomous drivingvehicle and an electronic device for vehicles.

Note that the description of these objects does not preclude theexistence of other objects. Note that one embodiment of the presentinvention does not have to achieve all these objects. Note that otherobjects will be apparent from the descriptions of the specification, thedrawings, the claims, and the like, and other objects can be derivedfrom the descriptions of the specification, the drawings, the claims,and the like.

Means for Solving the Problems

A technique for capturing a monochrome image and performing colorizationis particularly suitable for imaging at night. For example, the color ofan image captured at night is different from that of an image capturedduring the daytime because the amount of light at night is smaller thanthat of the daytime. In addition, in the case where imaging is performedusing infrared rays in a dark environment that needs imaging usinginfrared rays, a monochrome image is obtained. Furthermore, in imagingusing infrared rays, brightness greatly differs between a material thatreflects infrared rays and a material that absorbs infrared rays. Evenwhen these materials are placed in the same position, there is a depthproblem that the material that reflects infrared rays is seen in theforeground and the material that absorbs infrared rays is seen in thebackground.

In addition, when driving in a dark environment is assumed, in the casewhere the light of an oncoming car is illuminated, the visibility ofsurroundings becomes poor. In particular, the pupils of human eyesbecome small, so that the human eyes cannot perceive the surroundings.Accordingly, when image display by an imaging device that can freely setexposure is used accessorily, the ease of securing the safety of thesurroundings is increased. It is desirable that this image have highvisibility.

It is desirable that such an image be subjected to segmentationprocessing and an image of a human or a car be subjected to highlightingin addition to colorization. However, in the case of a distant image,the image is small and accuracy is decreased because the amount ofinformation is small. In particular, in the case where the imagingdevice is incorporated in a vehicle, an image is sometimes blurred dueto vibration or the like. In the case where the imaging device is usedas an in-vehicle camera, in particular, when vehicle speed is increased,distant information is needed.

As the vehicle speed is increased, a driver's viewing angle is narrowed;however, there is a contradiction in which a wide viewing angle isneeded to perform safe driving. In addition, it is not practical todecrease the vehicle speed in order to secure a wide viewing angle. Itis desirable to have an environment where safe driving can be performedregardless of whether the vehicle speed is low or high.

As the vehicle speed is increased, the driver's viewing angle isnarrowed; however, safety during the vehicle driving is increased byautomatically capturing an image of a region to be careful with a cameraand displaying the image for the driver to assist the driver.

On the other hand, it is possible to take an enlarged image byadjustment of a camera lens; however, it is difficult for the driver toperform adjustment such as enlargement or reduction of an opticalsystem, and it is also difficult to provide a mechanism where the driverchanges a lens direction to move a lens focus and changes an imagingdirection. In addition, even when an image of a distant portion iscaptured by a telephoto lens, an object is sometimes lost due tovibration or the like during driving. It is desirable that only animportant portion be automatically extracted and displayed withoutchanging the focal length or direction of the camera, independently ofthe driver. The important portion refers to, for example, a vehicle thatmoves on a distant road or a region where a pedestrian or the like ispresent.

A driving assistance system is desirable in which a specific region tobe careful is displayed and a burden on a driver is reduced duringvehicle driving. In addition, in the case where the region to be carefulis distant, a display system for enlarging the region is also desirable.

A structure of an invention disclosed in this specification is a drivingassistance system that includes an imaging device capable of capturing afirst monochrome image in a vehicle traveling direction, a first neuralnetwork for segmentation processing, a second neural network for depthestimation processing, a determination portion determining a center of aportion cut off from the first monochrome image on the basis of thesegmentation processing and the depth estimation processing, a thirdneural network for colorization processing of only a second cut-offmonochrome image, and a display device for enlargement of the secondmonochrome image subjected to the colorization processing.

In the above structure, the driving assistance system is incorporated ina vehicle and thus preferably uses a plurality of learned neuralnetworks and includes one or a plurality of storage portions in which aprogram for performing the plurality of learned neural networks. One ormore processors are incorporated in a vehicle to perform these neuralnetworks.

In addition, a driving assistance system includes a step of driving avehicle incorporating an imaging device, a step of capturing amonochrome image of a front of the vehicle during driving by the imagingdevice, a step of performing inference of a region of at least a sky, acar, and a road by performing segmentation processing on a monochromeimage including the distant region, a step of performing inference of aspecific distant region by performing depth estimation processing on themonochrome image including the distant region, a step of determining acenter of a portion cut off from the monochrome image on the basis ofthe segmentation processing and the depth estimation processing, a stepof extracting a rectangular region in which a center is a centralportion, inputting extracted data, and performing super-resolutionprocessing, a step of inputting an output result of the super-resolutionprocessing and performing colorization processing for highlighting of anobject included in the distant region with high accuracy, and a step ofperforming enlargement of the distant region subjected to colorizationprocessing. Note that the specific distant region refers to a regionincluding at least a road edge portion in a traveling direction.

Furthermore, the driving assistance system may further includes a stepof measuring driving speed of the vehicle in addition to the steps. Thesize of an image to be cut off can be changed depending on the drivingspeed of the vehicle. For example, the size of the rectangular region inwhich the center of the portion cut off from the monochrome image is thecentral portion can be determined by the driving speed of the vehicle. Acut-off area is large in the case of high vehicle speed compared to thecase of low vehicle speed. Accordingly, it is possible to make up forthe field of view of a driver that is narrowed by the speed.

One or more processors perform processing for reading and executing aprogram including any one or all of the steps. A program in which acomputer executes each step is stored in a storage portion in advance.In addition, there is no limitation to the processor, and a circuit thatachieves a function for performing any one or all of the steps (forexample, an FPGA circuit or an ASIC circuit) can also perform theprocessing for reading and executing the program.

In the above structure, the segmentation processing uses first neuralnetwork processing; the depth estimation processing uses second neuralnetwork processing; the super-resolution processing uses third neuralnetwork processing; and the colorization processing uses fourth neuralnetwork processing. As a training data set for learning of thesegmentation processing, MSCOCO, Cityscapes, or the like is used. Inaddition, as a training data set for learning of the depth estimationprocessing, KITTI or the like is used. As a training data set forlearning of the super-resolution processing, there is no particularlimitation, and not only a photograph but also an illustration may beused. As a training data set for learning of the colorizationprocessing, there is no particular limitation as long as the trainingdata set is a color training data set, and ImageNet or a color imagecaptured by a dashboard camera can be processed and used.

When the driving assistance system is specifically described, neuralnetwork processing is performed under a state in which color informationis reduced and the amount of captured information is reduced, thedistant region is cut off so that the amount of data is further reduced,only the distant region is subjected to colorization, and enlargement isperformed. The amount of data can be reduced by reduction of the colorinformation, and arithmetic processing in the neural network processingcan be made simple. In addition, when the amount of data can be reduced,the size of hardware capable of performing the neural network processingcan be made small. An imaging device that does not include a colorfilter has a wider dynamic range because not only color informationreduction but also no light reduction due to a color filter and ease ofsecuring the amount of light reaching a light-receiving sensor can beachieved.

In addition, a device or a vehicle including the imaging system and thedriving assistance system disclosed in this specification can also bereferred to as an image generation device. The image generation deviceselectively performs colorization processing on part of a monochromeimage with a wide dynamic range that is captured by an imaging devicewithout a color filter and performs enlargement.

Furthermore, an image colored by the colorization processing does nothave a natural color in many cases, and the image is highlighting, whichis easily recognized by the driver.

Moreover, there is no limitation to an imaging device without using acolor filter. In addition to the imaging device without using a colorfilter, a driving assistance system may be constructed by a combinationwith an imaging device including a color filter, or a combination withanother environment recognition unit, for example, a stereo camera, asonar, a multifocal multi-eye camera system, a LIDAR, a millimeter-waveradar, an infrared sensor (a TOF system), or the like. In distancemeasurement with the TOF system, a light source and a light detector (asensor or a camera) are used. A camera used in the TOF system isreferred to as a time-of-flight camera, and is also referred to as a TOFcamera. The TOF camera can obtain distance information between a lightsource emitting light and an object on the basis of time of flight ofreflected light of light delivered on the object.

Effect of the Invention

With the use of a plurality of neural networks, a region to be carefulthat is enlarged can be provided to a driver. Image display that mainlyassists the driver can be provided.

In addition, clear color display of a distant region in a vehicletraveling direction can be provided to the driver in an environment withan insufficient amount of light, for nightfall hours, for night hours,for early-morning hours, or when passing through a long tunnel, forexample; thus, there is an especially remarkable effect.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example of a flow chart showing oneembodiment of the present invention.

FIG. 2 is a diagram illustrating an example of a flow chart showing oneembodiment of the present invention.

FIG. 3 is a diagram illustrating an example of a flow chart showing oneembodiment of the present invention.

FIG. 4 is a block diagram illustrating one embodiment of the presentinvention.

FIG. 5 is a block diagram illustrating one embodiment of the presentinvention.

FIG. 6 is a block diagram illustrating a structure example of an imagingportion.

FIG. 7 is a diagram illustrating structure examples of a pixel block 200and a circuit 201.

FIG. 8 is a diagram illustrating a pixel structure example.

FIG. 9A to FIG. 9C are diagrams each illustrating a filter.

FIG. 10A is a diagram illustrating a pixel structure example. FIG. 10Bto FIG. 10D are diagrams each illustrating a structure example of aphotoelectric conversion device.

FIG. 11 is a cross-sectional view illustrating a structure example of animaging device.

FIG. 12A to FIG. 12C are cross-sectional views each illustrating atransistor structure example.

FIG. 13 is a cross-sectional view illustrating a structure example of animaging device.

FIG. 14 is a cross-sectional view illustrating a structure example of animaging device.

FIG. 15A1 to FIG. 15A3 are perspective views of a package in which animaging device is placed, and FIG. 15B1 to FIG. 15B3 are perspectiveviews of a module.

FIG. 16A is a diagram illustrating appearance of a vehicle, and FIG. 16Bis a schematic diagram showing the field of view of a driver positionedin the front of a vehicle seen from the inside of the vehicle.

FIG. 17 is an example of an applied product for which an imaging systemaccording to one embodiment of the present invention is employed.

MODE FOR CARRYING OUT THE INVENTION

Embodiments of the present invention will be described in detail belowwith reference to the drawings. Note that the present invention is notlimited to the following description, and it is readily understood bythose skilled in the art that modes and details of the present inventioncan be modified in various ways. In addition, the present inventionshould not be construed as being limited to the description of thefollowing embodiments.

Embodiment 1

In this embodiment, FIG. 1 illustrates an example of a flow of a drivingassistance system that selectively extracts a distant region to which adriver should pay attention from a monochrome video obtained by anin-vehicle solid-state imaging element, performs colorization on part ofthe extracted distant region, and provides an enlarged video to thedriver.

An imaging system that has a hardware structure including a solid-stateimaging element is placed in part of a vehicle (a hood, the inside ofthe vehicle, a roof, or the like) capable of shooting in a vehicletraveling direction (including a distant region), and is activated tostart continuous shooting.

In-vehicle hardware has a structure in which one or more processorsmainly control operation in each step. In the case where neural networkprocessing is performed, hardware including a storage portion such as amemory adequate for accumulating learning data and capable of adequatearithmetic processing is needed. The storage portion refers to a compactlarge-capacity storage device (for example, an SSD or a hard disk)capable of being incorporated in a vehicle. When a program stored in acompact large-capacity storage device is executed, processing in theflow chart illustrated in FIG. 1 can be achieved.

First, data acquisition starts (S1).

Monochrome image data is acquired using a solid-state imaging elementwithout a color filter (S2). Note that a plurality of solid-stateimaging elements arranged in a matrix direction is sometimes referred toas a pixel array. In addition, a display example 21 of a video capturedimmediately after Step S2 is illustrated on a left side in FIG. 1 . Notethat the display example 21 is illustrated for easy understanding and isnot actually displayed. The display example 21 is actually monochromeimage data that is converted into a signal format (JPEG (registeredtrademark) or the like).

Next, a distant region that is part of captured image data is extractedand cut off (S3). A distant region in a vehicle traveling direction is aregion that is most hardly recognized by the driver when driving in adark state, for example, in the evening, at night, or in the earlymorning. A solid-state imaging element without a color filter cancapture a video with a wider dynamic range than a solid-state imagingelement with a color filter; thus, an object in the distant region canbe captured as a video.

In this embodiment, the distant region in the vehicle travelingdirection is selectively extracted. As an extraction method, a portionis identified using depth estimation or data obtained by a depth sensormodule (a TOF camera or the like) that measures depth. In the case wheredepth estimation is used, neural network processing is performed. InStep S3, as an example in which the distant region is extracted, aregion surrounded by dotted lines in FIG. 1 is illustrated as a distantregion 22 to be extracted. When the distant region is cut off in StepS3, the amount of data to be used can be reduced.

In addition, in Step S3, size to be cut off may be determined using dataof a speed meter. The speed meter is a device that enables supplement ofsignals from a GPS and a GLONASS satellite. With the use of a GPS or thelike, the speed, position, or mileage of the vehicle during driving canbe measured.

Then, data is reduced to data of only the distant region 22 (S4). InStep S4, data other than the data cut off in S3 is deleted. Note thatthe original data may be stored in a dedicated storage device.

After that, colorization inference is performed on the data of only thedistant region 22 (S5). In this embodiment, the data reduced in Step S4is used as input data, convolutional processing is performed using a CPUor the like, and inference of an edge, a color, and the like isperformed for colorization.

In addition, in the case where monochrome image colorization is executedby software, a program or the like may be installed from a network, astorage medium, or a computer in which a program that constructssoftware is incorporated in hardware. A program stored in acomputer-readable storage medium such as a CD-ROM (Compact Disk ReadOnly Memory) is installed, and the program for monochrome imagecolorization is executed. Processing by the program is not necessarilyperformed in order or on the time series, and may be performed inparallel, for example.

Furthermore, a program of software executing an inference program forneural network processing used for depth estimation or colorizationinference can be described in a variety of programing languages such asPython, Go, Perl, Ruby, Prolog, Visual Basic, C, C++, Swift, Java(registered trademark), and NET. Moreover, an application may be madeusing a framework such as Chainer (it can be used with Python), Caffe(it can be used with Python and C++), and TensorFlow (it can be usedwith C, C++, and Python). For example, the algorithm of LSTM isprogrammed with Python, and a CPU (Central Processing Unit) or a GPU(Graphics Processing Unit) is used. A chip in which a CPU and a GPU areintegrated is sometimes referred to as an APU (Accelerated ProcessingUnit), and this APU chip can also be used. Alternatively, an ICincorporating an AI system (also referred to as an inference chip) maybe used. The IC incorporating an AI system is sometimes referred to as acircuit performing neural network calculation (a microprocessor).

In this embodiment, inference is performed inside the vehicle; thus, afeature value learned in advance (a colorization weight) is used. Bystoring the learned feature value in the storage device and performingarithmetic operation, it is possible to output data at a level similarto that when not using the learned feature value. Since data reduced inadvance is used, a load on arithmetic processing is reduced. Note thatFIG. 1 illustrates a colorization image 23 as a display example aftercolorization.

The thus obtained colorization image 23 uses an imaging element withouta color filter and is based on a monochrome image with wide dynamicrange; thus, even in the case where a conventional imaging element witha color filter cannot perform identification because of a small amountof light, identifiable colorization image data can be obtained.

Finally, the colorization image 23 is enlarged on a display portion of adisplay device as a highlighted image 24 (S6). The highlighted image canbe obtained by enlargement. However, colorization is further performedusing colorization inference in this embodiment, and an image that isdifferent from an actual video is obtained though the image has a colorclose to a natural color. Accordingly, the highlighted image can bemade. Note that either Step S6 or Step S5 may be performed first becausethe result is the same even when the sequence of Step S6 and Step S5 isreversed.

Acquisition of the highlighted image is repeated. Repeated acquisitionalso enables real-time display of the distant region as the highlightedimage.

The driving assistance system can clearly achieve imaging in acomparatively dark place and can provide the driver with a region towhich the driver needs to pay attention as a highlighted image. Thedriving assistance system can particularly prevent accidents in a darkplace with little lighting, such as in the evening or at night.

In addition, when a camera for imaging of vehicle surroundings or aradar is combined with an ECU (Electronic Control Unit) for imageprocessing or the like, the driving assistance system can also beapplied to a vehicle capable of semi-autonomous driving or a vehiclecapable of fully autonomous driving. A vehicle using an electric motorincludes a plurality of ECUs, and engine control and the like areperformed by the ECUs. The ECU includes a microcomputer. The ECU isconnected to a CAN (Controller Area Network) provided in the electricvehicle. The CAN is a type of a serial communication standard used as anin-vehicle LAN. For the ECU, a CPU or a GPU is used. For example, astructure can be employed in which a solid-state imaging element withouta color filter is used as one of a plurality of cameras (dashboardcameras, rear cameras, and the like) incorporated in an electric vehicleso that part of an obtained monochrome image can be extracted, inferencecan be performed in the ECU through the CAN, a colorization image can becreated, and a highlighted image can be displayed on a display portionof an in-vehicle display device by enlargement.

Furthermore, in this embodiment, in the case where a portion to beextracted is identified using data obtained by a depth sensor module (aTOF camera or the like) that measures depth as a method for selectivelyextracting a distant region, colorization inference is performed byperforming neural network processing once. In that case, since neuralnetwork processing is performed only once, arithmetic processing can besmall-scale arithmetic processing, which has an advantage of a smallload on a CPU.

Alternatively, in the case where depth estimation is used as a methodfor selectively extracting a distant region, first neural networkprocessing is performed for depth estimation, and second neural networkprocessing is performed for colorization inference. In that case, it ispossible to eliminate the need for providing a depth sensor module thatmeasures depth. In addition, although neural network processing isperformed twice, a load on arithmetic operation is reduced by reductionof input data of neural network processing for colorization.

Embodiment 2

In this embodiment, an example in which a method for extracting adistant region is different from that in Embodiment 1 will be described.Many other portions are the same as those in Embodiment 1; thus,detailed description thereof will be omitted here.

FIG. 2 illustrates an example of a flow of a driving assistance system.Note that the same reference numerals are used for steps that are thesame as those in the flow chart shown in FIG. 1 in Embodiment 1.

First, data acquisition starts (S1).

Monochrome image data is acquired using a solid-state imaging elementwithout a color filter (S2).

Then, in order to select a distant region that is part of captured imagedata, depth estimation is performed using monochrome image data (S21A).In addition, segmentation inference is also performed using themonochrome image data (21B).

Embodiment 1 also illustrates an example of depth estimation. In thisembodiment, inference is performed inside a vehicle; thus, a featurevalue learned in advance (a depth weight) is used.

Segmentation inference is also referred to as segmentation. Note thatsegmentation refers to processing for identifying what object each pixelof an input image represents. This is also referred to as semanticsegmentation. Software for generating a plurality of image segments foruse in image analysis is executed by neural network processing.Specifically, segmentation is performed on the basis of learned contentby using U-net, FCRN (Fully Convolutional Residual Networks), or thelike that is a type of image processing and a convolutional neuralnetwork (CNN). Note that the label of segmentation is differentiated bya vehicle, a sky, a plant, a ground, a human, a building, or the like.In addition, in this embodiment, inference is performed inside thevehicle; thus, a feature value learned in advance (a segmentationweight) is used.

Step S21A and Step S21B may be sequentially performed or may beperformed in parallel.

Next, regions of a sky and a road obtained by segmentation inference arefocused on, and the coordinates of an upper end portion of the roadwhere a distance between the sky and the road (a space between a lowerend portion of the sky and an upper end portion of the road) is theshortest is extracted in a segmentation image. This is because thecircumference of a driving lane end portion corresponds to a portionwhere the space between the sky and the road in the image is theshortest, as shown in the display example 21 in FIG. 1 .

The number of portions where the coordinates on the road in which thedistance between the sky and the road is the shortest is not limited toone in the segmentation image, and the number of such portions issometimes more than one.

In the case where the number of portions where the coordinates on theroad in which the distance between the sky and the road is the shortestis more than one in the segmentation image, the most distant coordinatesare selected among them (S22).

From a region of the ground obtained in Step S21B, a portion with themaximum depth in the region is extracted from the results in Step 21A.By identifying a driving lane end portion in a monochrome image, acentral portion of the distant region to be cut off later can bedetermined.

Then, data is reduced to data of only the distant region (S23).

After that, colorization inference is performed on the data of only thedistant region (S24). In this embodiment, inference is performed insidethe vehicle; thus, a feature value learned in advance (a colorizationweight) is used.

Finally, the colorization image is enlarged on a display portion of adisplay device as a highlighted image (S25).

Acquisition of the highlighted image is repeated.

In this embodiment, as a method for selectively extracting a distantregion, first neural network processing is performed for depthestimation, second neural network processing is performed forsegmentation inference, and third neural network processing is performedfor colorization inference. A feature value learned in advance is usedin each of these inferences; thus, arithmetic operation is performedinside the vehicle.

This embodiment can be freely combined with Embodiment 1.

Embodiment 3

In this embodiment, an example in which a method for extracting adistant region is different from that in Embodiment 2 will be described.Many other portions are the same as those in Embodiment 1 or Embodiment2; thus, detailed description thereof will be omitted here.

FIG. 3 illustrates an example of a flow of a driving assistance system.Note that the same reference numerals are used for steps that are thesame as those in the flow chart shown in FIG. 1 in Embodiment 1 and FIG.2 in Embodiment 2.

Monochrome image data is acquired using a solid-state imaging elementwithout a color filter (S2).

Then, in order to select a distant region that is part of captured imagedata, depth estimation is performed using monochrome image data (S21A).In addition, segmentation inference is also performed using themonochrome image data (21B).

Next, the distant region is selected (S22).

In addition, vehicle speed data is acquired (S26). The speed meter is adevice that enables supplement of signals from a GPS and a GLONASSsatellite. With the use of a GPS or the like, the speed, position, ormileage of a vehicle during driving can be measured. Furthermore, anumerical value that is obtained by a typical speed meter may be used asthe vehicle speed data. Note that the timing of Step S26 is notparticularly limited and may be any time between the start of dataacquisition and Step S22.

Then, the cut-off size of distant size is determined (S27). The cut-offsize is determined using data of the speed meter that is obtained inStep S26. For example, a structure is employed in which a cut-off areais large in the case of high speed compared to the case of low speed.

Then, data is reduced to data of only the distant region (S28).

After that, super-resolution inference is performed on the data of onlythe distant region (S29). Note that super-resolution processing refersto image processing for generating a high-resolution image from alow-resolution image. Super-resolution processing may be repeated morethan once. For creation of a learning model for determining a colorboundary, by mixing of not only a color picture image but also a colorillustration (animation) image as training data, colored image data witha clear color boundary can be obtained. Accordingly, an illustration(animation) image is preferably mixed into training data duringlearning, and a super-resolution weight to be used as a weightcoefficient of neural network processing is calculated. In thisembodiment, inference is performed inside the vehicle; thus, a featurevalue learned in advance (a super-resolution weight) is used.

Then, colorization inference is performed (S30). In this embodiment,inference is performed inside the vehicle; thus, a feature value learnedin advance (a colorization weight) is used.

The sequence of performing colorization inference after super-resolutioninference is also important. It is preferable that arithmetic operationbe performed using monochrome image data and colorization be performedas final image processing. The monochrome image data has a smaller dataamount than image data that has color information, and a load onprocessing capacity of an arithmetic unit can be reduced.

Finally, the colorization image is enlarged on a display portion of adisplay device as a highlighted image (S31).

Acquisition of the highlighted image is repeated.

In this embodiment, as a method for selectively extracting a distantregion, first neural network processing is performed for depthestimation, second neural network processing is performed forsegmentation inference, third neural network processing is performed forsuper-resolution inference, and fourth neural network processing isperformed for colorization inference.

A feature value learned in advance is used in each of these inferences;thus, arithmetic operation is performed inside the vehicle. A wide areais enlarged and subjected to super-resolution processing in the case ofhigh speed; thus, clear highlighting can be obtained.

This embodiment can be freely combined with Embodiment 1 or Embodiment2.

Embodiment 4

In this embodiment, an example of a block diagram of a drivingassistance system 31 that executes the flow chart in Embodiment 2 isdescribed. FIG. 4 is a block diagram of the driving assistance system31, and the following description is made with reference to FIG. 4 .

A data acquisition device 10 is a semiconductor chip that includes asolid-state imaging element 11 and a memory portion 14 and does notinclude a color filter. The data acquisition device 10 includes anoptical system such as a lens. Note that the optical system is notparticularly limited as long as imaging characteristics are known, andan optical system with any structure may be employed.

For example, as the data acquisition device 10, one semiconductor chipwhere a back-illuminated CMOS image sensor chip, a DRAM chip, and alogic circuit chip are stacked may be used. In addition, onesemiconductor chip where a back-illuminated CMOS image sensor chip and alogic circuit chip including an analog/digital converter circuit arestacked may be used. In that case, the memory portion 14 is an SRAM.Furthermore, chips to be stacked are stacked using a known bondingtechnology to achieve electrical connection.

The memory portion 14 is a circuit that stores digital data afterconversion and has a structure in which data is stored before input toneural network portions 16 a, 16 b, and 16 c; however, the presentinvention is not limited to this structure.

The neural network portions 16 a, 16 b, and 16 c are achieved bysoftware calculation with a microcontroller. The microcontroller isobtained by incorporating a computer system into one integrated circuit(IC). When the calculation scale or data to be handled is large, aplurality of ICs are combined to form the neural network portions 16 a,16 b, and 16 c. A learning device includes at least the plurality ofICs. In addition, it is preferable to use a microcontrollerincorporating Linux (registered trademark) that enables use of freesoftware because the total cost of forming the neural network portions16 a, 16 b, and 16 c can be reduced. Furthermore, another OS (operatingsystem) may be used without being limited to Linux (registeredtrademark).

Learning of the neural network portions 16 a, 16 b, and 16 c illustratedin FIG. 4 is described below. In this embodiment, learning is performedin advance, and neural network processing is performed utilizing aweight. Training data for learning can be stored in storage portions 18a, 18 b, and 18 c so that learning can also be performed as appropriate.

In Embodiment 2, only a distant region is selected based on outputresults of the neural network portions 16 a and 16 b so that data isreduced. A data extraction portion 17 can also be referred to as a datacontrol portion that selects only a distant region and reduces data.Data extracted by the data extraction portion 17 is input to the neuralnetwork portion 16 c so that colorization is performed.

Data output from the neural network portion 16 c has information on anedge and a color of an object that is related to an area for only thedistant region and is input to a display device 15. The display device15 includes a display portion 19 and forms a signal showing a videoincluding enlargement in accordance with a gray scale level that can bedisplayed or a screen size.

In addition, a fellow passenger's portable information terminal (asmartphone or the like) can be the display device 15, and a displayportion of the portable information terminal can be the display portion19. In that case, a transmission/reception portion for transmitting anoutput from the neural network portion 16 c to the fellow passenger'sportable information terminal needs to be incorporated in a vehicle. Itis needless to say that, without limitation to the fellow passenger, adriver's portable information terminal (a smartphone or the like) may beplaced in a hood or the like so that the driver can see the video.

The driving assistance system 31 can particularly prevent accidents in adark place with little lighting, such as in the evening or at night.

This embodiment can be freely combined with Embodiment 1, Embodiment 2,or Embodiment 3.

Embodiment 5

This embodiment illustrates a modification example where part ofEmbodiment 4 is changed. Note that description is made using the samereference numerals for the same portions as those in Embodiment 4.Structures of components below the memory portion 14 are the same asthose in Embodiment 4; thus, description thereof is omitted here.

A structure example of an imaging system 41 is described with referenceto a block diagram illustrated in FIG. 5 .

The data acquisition device 10 is a semiconductor chip that includes thesolid-state imaging element 11 and the analog arithmetic circuit 12 anddoes not include a color filter. The data acquisition device 10 includesan optical system such as a lens. Note that the optical system is notparticularly limited as long as imaging characteristics are known, andan optical system with any structure may be employed.

In addition, a transistor using a metal oxide formed over a silicon chipof a solid-state imaging element formed using a silicon substrate(hereinafter, an OS transistor) or the like can be used for the analogarithmetic circuit 12.

An A/D circuit 13 (also referred to as an A/D converter) illustrates ananalog-to-digital conversion circuit and converts analog data outputfrom the data acquisition device 10 into digital data. Note that ifneeded, an amplifier circuit may be provided between the dataacquisition device 10 and the A/D circuit 13 so that an analog signal isamplified before conversion into the digital data.

The memory portion 14 is a circuit that stores digital data afterconversion and has a structure in which data is stored before input tothe neural network portions 16 a, 16 b, and 16 c; however, the presentinvention is not limited to this structure. Although it depends on theamount of data output from the data acquisition device or the dataprocessing capacity of an image processing device, a structure may beemployed in which small-scale data output from the A/D circuit 13 isdirectly input to the neural network portions 16 a, 16 b, and 16 c,without storing the small-scale data in the memory portion 14.

In this embodiment, owing to the analog arithmetic circuit illustratedin FIG. 5 , part of arithmetic operation that is performed in commonamong the neural network portions 16 a, 16 b, and 16 c can be performedin advance. The use of the imaging system 41 illustrated in FIG. 5 canreduce the number of arithmetic operations performed in the neuralnetwork portions 16 a, 16 b, and 16 c.

This embodiment can be freely combined with Embodiment 1, Embodiment 2,Embodiment 3, or Embodiment 4.

Embodiment 6

In this embodiment, a structure example where the data acquisitiondevice 10 is part of the structure of the imaging system 41 is describedbelow. FIG. 6 is a block diagram illustrating the imaging system 41.

The imaging system 41 includes a pixel array 300, a circuit 201, acircuit 301, a circuit 302, a circuit 303, a circuit 304, a circuit 305,and a circuit 306. Note that each of the structures of the circuit 201and the circuit 301 to the circuit 306 is not limited to a singlecircuit structure and is sometimes composed of a combination of aplurality of circuits. Alternatively, any of the plurality of circuitsdescribed above may be combined. Furthermore, a circuit other than theabove circuits may be connected.

The pixel array 300 has an imaging function and an arithmetic function.The circuit 201 and the circuit 301 each have an arithmetic function.The circuit 302 has an arithmetic function or a data conversionfunction. The circuit 303, the circuit 304, and the circuit 306 eachhave a selection function. The circuit 303 is electrically connected toa pixel block 200 through a wiring 424. The circuit 304 is electricallyconnected to the pixel block 200 through a wiring 423. The circuit 305has a function of supplying a potential for product-sum operation to apixel. As a circuit having a selection function, a shift register, adecoder, or the like can be used. The circuit 306 is electricallyconnected to the pixel block 200 through a wiring 413. Note that thecircuit 301 and the circuit 302 may be provided outside.

The pixel array 300 includes a plurality of pixel blocks 200. Asillustrated in FIG. 7 , the pixel block 200 includes a plurality ofpixels 400 arranged in a matrix, and each of the pixels 400 iselectrically connected to the circuit 201 through a wiring 412. Notethat the circuit 201 can also be provided in the pixel block 200.

Furthermore, the pixels 400 are electrically connected to adjacentpixels 400 through transistors 450 (a transistor 450 a to a transistor450 f). The functions of the transistors 450 are described later.

The pixels 400 can acquire image data and generate data obtained byadding the image data and a weight coefficient. Note that the number ofpixels included in the pixel block 200 is 3×3 in an example illustratedin FIG. 7 but is not limited to this. For example, the number of pixelscan be 2×2, 4×4, or the like. Alternatively, the number of pixels in ahorizontal direction and the number of pixels in a vertical directionmay differ from each other. Furthermore, some pixels may be shared byadjacent pixel blocks. Although ten transistors 450 (transistors 450 ato 450 j) are provided between the pixels 400 in the examplesillustrated in FIG. 7 , the number of transistors 450 may be furtherincreased. In addition, in the transistors 450 g to 450 j, sometransistors may be omitted so that a parallel path is canceled. Wirings413 g to 413 j are respectively connected to the transistors 450 g to450 j as gates.

The pixel block 200 and the circuit 201 can operate as a product-sumoperation circuit.

As illustrated in FIG. 8 , the pixel 400 can include a photoelectricconversion device 401, a transistor 402, a transistor 403, a transistor404, a transistor 405, a transistor 406, and a capacitor 407.

One electrode of the photoelectric conversion device 401 is electricallyconnected to one of a source and a drain of the transistor 402. Theother of the source and the drain of the transistor 402 is electricallyconnected to one of a source and a drain of the transistor 403, a gateof the transistor 404, and one electrode of the capacitor 407. One of asource and a drain of the transistor 404 is electrically connected toone of a source and a drain of the transistor 405. The other electrodeof the capacitor 407 is electrically connected to one of a source and adrain of the transistor 406.

The other electrode of the photoelectric conversion device 401 iselectrically connected to a wiring 414. The other of the source and thedrain of the transistor 403 is electrically connected to a wiring 415.The other of the source and the drain of the transistor 405 iselectrically connected to the wiring 412. The other of the source andthe drain of the transistor 404 is electrically connected to a GNDwiring or the like. The other of the source and the drain of thetransistor 406 is electrically connected to a wiring 411. The otherelectrode of the capacitor 407 is electrically connected to a wiring417.

A gate of the transistor 402 is electrically connected to a wiring 421.A gate of the transistor 403 is electrically connected to a wiring 422.A gate of the transistor 405 is electrically connected to the wiring423. A gate of the transistor 406 is electrically connected to thewiring 424.

Here, a point where the other of the source and the drain of thetransistor 402, the one of the source and the drain of the transistor403, the one electrode of the capacitor 407, and the gate of thetransistor 404 are electrically connected is referred to as a node FD.Furthermore, a point where the other electrode of the capacitor 407 andthe one of the source and the drain of the transistor 406 areelectrically connected is referred to as a node FDW.

The wiring 414 and the wiring 415 can each have a function of a powersupply line. For example, the wiring 414 can function as a highpotential power supply line, and the wiring 415 can function as a lowpotential power supply line. The wiring 421, the wiring 422, the wiring423, and the wiring 424 can function as signal lines that control theconduction of the respective transistors. The wiring 411 can function asa wiring for supplying a potential corresponding to a weight coefficientto the pixel 400. The wiring 412 can function as a wiring thatelectrically connects the pixel 400 and the circuit 201. The wiring 417can function as a wiring that electrically connects the other electrodeof the capacitor 407 of the pixel 400 and the other electrode of thecapacitor 407 of another pixel 400 through the transistor 450 (see FIG.7 ).

Note that an amplifier circuit or a gain control circuit may beelectrically connected to the wiring 412.

As the photoelectric conversion device 401, a photodiode can be used.There is no limitation on types of photodiodes, and it is possible touse a Si photodiode in which a photoelectric conversion layer containssilicon, an organic photodiode in which a photoelectric conversion layerincludes an organic photoconductive film, or the like. Note that inorder to increase light detection sensitivity under low illuminanceconditions, an avalanche photodiode is preferably used.

The transistor 402 can have a function of controlling the potential ofthe node FD. The transistor 403 can have a function of initializing thepotential of the node FD. The transistor 404 can have a function ofcontrolling current fed by the circuit 201 in accordance with thepotential of the node FD. The transistor 405 can have a function ofselecting a pixel. The transistor 406 can have a function of supplyingthe potential corresponding to the weight coefficient to the node FDW.

In the case where an avalanche photodiode is used as the photoelectricconversion device 401, high voltage is sometimes applied and thus atransistor with high breakdown voltage is preferably used as atransistor connected to the photoelectric conversion device 401. As thetransistor with high breakdown voltage, a transistor using a metal oxidein its channel formation region (hereinafter an OS transistor) or thelike can be used, for example. Specifically, an OS transistor ispreferably employed as the transistor 402.

An OS transistor also has a feature of extremely low off-state current.When OS transistors are used as the transistor 402, the transistor 403,and the transistor 406, the charge retention period of the node FD andthe node FDW can be lengthened greatly. Therefore, a global shutter modein which charge accumulation operation is performed in all the pixels atthe same time can be employed without complicating the circuit structureand the operation method. Furthermore, while image data is retained atthe node FD, arithmetic operation using the image data can be performedmore than once.

Meanwhile, it is sometimes desirable that the transistor 404 haveexcellent amplifying characteristics. In addition, a transistor havinghigh mobility capable of high-speed operation is sometimes preferablyused as the transistor 406. Accordingly, transistors using silicon intheir channel formation regions (hereinafter Si transistors) may beemployed as the transistor 404 and the transistor 406.

Note that without limitation to the above, an OS transistor and a Sitransistor may be freely employed in combination. Alternatively, all thetransistors may be OS transistors. Alternatively, all the transistorsmay be Si transistors. Examples of Si transistors include a transistorincluding amorphous silicon, a transistor including crystalline silicon(microcrystalline silicon, low-temperature polysilicon, or singlecrystal silicon), and the like.

The potential of the node FD in the pixel 400 is determined by thepotential obtained by adding a reset potential supplied from the wiring415 and a potential (image data) generated by photoelectric conversionby the photoelectric conversion device 401. Alternatively, the potentialof the node FD in the pixel 400 is determined by capacitive coupling ofthe potential corresponding to a weight coefficient supplied from thewiring 411. Thus, current corresponding to data in which a given weightcoefficient is added to the image data can flow through the transistor405.

Note that the circuit structures of the pixel 400 described above areexamples, and the photoelectric conversion operation can also beperformed with other circuit structures.

As illustrated in FIG. 7 , the pixels 400 are electrically connected toeach other through the wiring 412. The circuit 201 can performarithmetic operation using the sum of currents flowing through thetransistors 404 of the pixels 400.

The circuit 201 includes a capacitor 202, a transistor 203, a transistor204, a transistor 205, a transistor 206, and a resistor 207.

One electrode of the capacitor 202 is electrically connected to one of asource and a drain of the transistor 203. The one of the source and thedrain of the transistor 203 is electrically connected to a gate of thetransistor 204. One of a source and a drain of the transistor 204 iselectrically connected to one of a source and a drain of the transistor205. The one of the source and the drain of the transistor 205 iselectrically connected to one of a source and a drain of the transistor206. One electrode of the resistor 207 is electrically connected to theother electrode of the capacitor 202.

The other electrode of the capacitor 202 is electrically connected tothe wiring 412. The other of the source and the drain of the transistor203 is electrically connected to a wiring 218. The other of the sourceand the drain of the transistor 204 is electrically connected to awiring 219. The other of the source and the drain of the transistor 205is electrically connected to a reference power supply line such as a GNDwiring. The other of the source and the drain of the transistor 206 iselectrically connected to a wiring 212. The other electrode of theresistor 207 is electrically connected to a wiring 217.

The wiring 217, the wiring 218, and the wiring 219 can each have afunction of a power supply line. For example, the wiring 218 can have afunction of a wiring that supplies a potential dedicated to reading. Thewiring 217 and the wiring 219 can function as high potential powersupply lines. The wiring 213, the wiring 215, and the wiring 216 canfunction as signal lines for controlling the conduction of therespective transistors. The wiring 212 is an output line and can beelectrically connected to the circuit 301 illustrated in FIG. 6 , forexample.

The transistor 203 can have a function of resetting the potential of thewiring 211 to the potential of the wiring 218. The wiring 211 is awiring that is electrically connected to the one electrode of thecapacitor 202, the one of the source and the drain of the transistor203, and the gate of the transistor 204. The transistor 204 and thetransistor 205 can have a function of a source follower circuit. Thetransistor 206 can have a function of controlling reading. The circuit201 has a function of a correlated double sampling circuit (a CDScircuit) and can be replaced with a circuit having the function andanother structure.

In one embodiment of the present invention, offset components other thanthe product of image data (X) and a weight coefficient (W) areeliminated, and an objective WX is extracted. WX can be calculated usingdata obtained when imaging is performed, data obtained when imaging isnot performed, and data obtained by adding weights to the respectivedata.

The total amount of currents (I_(p)) flowing through the pixels 400 whenimaging is performed is kΣ(X−V_(th))², and the total amount of currents(I_(p)) flowing through the pixels 400 when weights are added iskΣ(W+X−V_(th))². In addition, the total amount of currents (I_(ref))flowing through the pixels 400 when imaging is not performed iskΣ(0−V_(th))², and the total amount of currents (I_(ref)) flowingthrough the pixels 400 when weights are added is kΣ(W−V_(th))². Here, kis a constant and V_(th) is the threshold voltage of the transistor 405.

First, a difference (data A) between the data obtained when imaging isperformed and the data obtained by adding a weight to the data iscalculated. The difference iskΣ((X−V_(th))²−(W+X−V_(th))²)=kΣ(−W²−2W·X+2 W·V_(th)).

Next, a difference (data B) between the data obtained when imaging isnot performed and the data obtained by adding a weight to the data iscalculated. The difference is kΣ((0−V_(th))²−(W−V_(th))²)=kΣ(−W²+2W·V_(th)).

Then, a difference between the data A and the data B is calculated. Thedifference is kΣ(−W²−2W·X+2 W·V_(th)−(−W²+2 W·V_(th)))=kΣ(−2W·X). Thatis, offset components other than the product of the image data (X) andthe weight coefficient (W) can be eliminated.

The circuit 201 can read the data A and the data B. Note that thecalculation of the difference between the data A and the data B can beperformed by the circuit 301, for example.

Here, the weights supplied to the entire pixel block 200 function as afilter. As the filter, a convolutional filter of a convolutional neuralnetwork (CNN) can be used, for example. Alternatively, an imageprocessing filter such as an edge extraction filter can be used. Asexamples of the edge extraction filter, a Laplacian filter illustratedin FIG. 9A, a Prewitt filter illustrated in FIG. 9B, a Sobel filterillustrated in FIG. 9C, and the like can be given.

In the case where the number of pixels 400 included in the pixel block200 is 3×3, elements of the edge extraction filter can be assigned andsupplied as weights to the pixels 400. As described above, to calculatethe data A and the data B, data obtained when imaging is performed, dataobtained when imaging is not performed, and data obtained by addingweights to the respective data can be utilized for the calculation.Here, the data obtained when imaging is performed and the data obtainedwhen imaging is not performed are data to which weights are not addedand can also be referred to as data obtained by adding a weight 0 to allthe pixels 400.

The edge extraction filters illustrated as examples in FIG. 9A to FIG.9C are filters where the sum (ΣΔW/N, where Nis the number of elements)of elements (weights: ΔW) is 0. Therefore, without additional operationof supplying ΔW=0 from another circuit, the operation of obtaining ΣΔW/Nenables data corresponding to the data obtained by adding ΔW=0 to allthe pixels 400 to be acquired.

This operation corresponds to turning on the transistors 450 (thetransistors 450 a to 450 f) provided between the pixels 400 (see FIG. 7). By turning on the transistors 450, the node FDW in each of the pixels400 is short-circuited through the wiring 417. At this time, chargeaccumulated in the node FDW in each of the pixels 400 is redistributed,and in the case where the edge extraction filters illustrated asexamples in FIG. 9A to FIG. 9C are used, the potential of the node FDW(ΔW) becomes 0 or substantially 0. Thus, the data corresponding to thedata obtained by adding ΔW=0 can be acquired.

Note that in the case of rewriting weights (ΔW) by supplying charge froma circuit outside the pixel array 300, it takes time to completerewriting owing to the capacitance of the long-distance wiring 411 orthe like. In contrast, the pixel block 200 is a minute region, and thewiring 417 has a short distance and small capacitance. Therefore,weights (ΔW) can be rewritten at high speed by the operation ofredistributing charge accumulated in the nodes FDW in the pixel block200.

In the pixel block 200 illustrated in FIG. 7 , a structure where thetransistor 450 a to the transistor 450 f are electrically connected todifferent gate lines (a wiring 413 a to a wiring 413 f) is illustrated.With this structure, the conductions of the transistor 450 a to thetransistor 450 f can be controlled independently of each other, and theoperation of obtaining ΣΔW/N can be performed selectively.

For example, in the case of using a filter illustrated in FIG. 9B, FIG.9C, or the like, there are some pixels where ΔW=0 is initially supplied.Assuming that ΣΔW/N=0, the pixels where ΔW=0 is supplied may be excludedfrom the target of summation. The exclusion of the pixels eliminates theneed of supplying a potential for operating some of the transistors 450a to the transistor 450 f, which can reduce power consumption.

Product-sum operation result data output from the circuit 201 issequentially input to the circuit 301. The circuit 301 may have avariety of arithmetic functions in addition to the above-describedfunction of calculating the difference between the data A and the dataB. For example, the circuit 301 can have a structure similar to that ofthe circuit 201. Alternatively, the function of the circuit 301 may bereplaced by software processing.

In addition, the circuit 301 may include a circuit that performsarithmetic operation of an activation function. A comparator circuit canbe used as the circuit, for example. A comparator circuit outputs aresult of comparing input data and a set threshold as binary data. Inother words, the pixel blocks 200 and the circuit 301 can operate assome components of a neural network.

Data output from the circuit 301 is sequentially input to the circuit302. The circuit 302 can have a structure including a latch circuit, ashift register, and the like, for example. With this structure,parallel-serial conversion can be performed and data input in parallelcan be output to a wiring 311 as serial data.

Pixel Structure Example

FIG. 10A is a diagram illustrating a structure example of the pixel 400.The pixel 400 can have a stack structure of a layer 561 and a layer 563.

The layer 561 includes the photoelectric conversion device 401. Thephotoelectric conversion device 401 can include a layer 565 a and alayer 565 b, as illustrated in FIG. 10B. Note that a layer may berephrased as a region, depending on the case.

The photoelectric conversion device 401 illustrated in FIG. 10B is apn-junction photodiode; for example, a p-type semiconductor can be usedfor the layer 565 a, and an n-type semiconductor can be used for thelayer 565 b. Alternatively, an n-type semiconductor may be used for thelayer 565 a, and a p-type semiconductor may be used for the layer 565 b.

The pn junction photodiode can be formed typically using single crystalsilicon.

In addition, the photoelectric conversion device 401 included in thelayer 561 may have a stack of a layer 566 a, a layer 566 b, a layer 566c, and a layer 566 d, as illustrated in FIG. 10C.

The photoelectric conversion device 401 illustrated in FIG. 10C is anexample of an avalanche photodiode; the layer 566 a and the layer 566 dcorrespond to electrodes, and the layer 566 b and the layer 566 ccorrespond to a photoelectric conversion portion.

The layer 566 a is preferably a low-resistance metal layer or the like.For example, aluminum, titanium, tungsten, tantalum, silver, or a stackthereof can be used.

A conductive layer having a high light-transmitting property withrespect to visible light is preferably used as the layer 566 d. Forexample, indium oxide, tin oxide, zinc oxide, indium tin oxide, galliumzinc oxide, indium gallium zinc oxide, graphene, or the like can beused. Note that a structure in which the layer 566 d is omitted can alsobe employed.

The layer 566 b and the layer 566 c of the photoelectric conversionportion can be used to form a pn-junction photodiode containing aselenium-based material in a photoelectric conversion layer, forexample. A selenium-based material, which is a p-type semiconductor, ispreferably used for the layer 566 b, and gallium oxide or the like,which is an n-type semiconductor, is preferably used for the layer 566c.

A photoelectric conversion device containing a selenium-based materialhas characteristics of high external quantum efficiency with respect tovisible light. In the photoelectric conversion device, electrons can begreatly amplified with respect to the amount of incident light byutilizing avalanche multiplication. In addition, a selenium-basedmaterial has a high light-absorption coefficient and thus has advantagesin production; for example, a photoelectric conversion layer can bemanufactured using a thin film. A thin film of a selenium-based materialcan be formed by a vacuum evaporation method, a sputtering method, orthe like.

As a selenium-based material, crystalline selenium such as singlecrystal selenium or polycrystalline selenium, amorphous selenium, acompound of copper, indium, and selenium (CIS), a compound of copper,indium, gallium, and selenium (CIGS), or the like can be used.

An n-type semiconductor is preferably formed using a material with awide band gap and a light-transmitting property with respect to visiblelight. For example, zinc oxide, gallium oxide, indium oxide, tin oxide,a mixed oxide thereof, or the like can be used. In addition, thesematerials have a function of a hole-injection blocking layer, so thatdark current can be decreased.

In addition, the photoelectric conversion device 401 included in thelayer 561 may have a stack of a layer 567 a, a layer 567 b, a layer 567c, a layer 567 d, and a layer 567 e, as illustrated in FIG. 10D. Thephotoelectric conversion device 401 illustrated in FIG. 10D is anexample of an organic photoconductive film; the layer 567 a is a lowerelectrode, the layer 567 e is an upper electrode having alight-transmitting property, and the layer 567 b, the layer 567 c, andthe layer 567 d correspond to a photoelectric conversion portion.

One of the layer 567 b and the layer 567 d of the photoelectricconversion portion can be a hole-transport layer. In addition, the otherof the layer 567 b and the layer 567 d can be an electron-transportlayer. Furthermore, the layer 567 c can be a photoelectric conversionlayer.

For the hole-transport layer, molybdenum oxide or the like can be used,for example. For the electron-transport layer, for example, fullerenesuch as C₆₀ or C₇₀, a derivative thereof, or the like can be used.

As the photoelectric conversion layer, a mixed layer of an n-typeorganic semiconductor and a p-type organic semiconductor (a bulkheterojunction structure) can be used.

For example, the layer 563 illustrated in FIG. 10A includes a siliconsubstrate. The silicon substrate can be provided with a Si transistor orthe like. With the use of the Si transistor, the pixel 400 can beformed. In addition, a circuit 201 and a circuit 301 to a circuit 306that are illustrated in FIG. 6 can be formed.

Next, a stack structure of the imaging device is described using across-sectional view. Note that components such as insulating layers andconductive layers are described below as examples, and other componentsmay be further included. Alternatively, some components described belowmay be omitted. In addition, a stack structure described below can beformed by a bonding process, a polishing process, or the like as needed.

An imaging device with a structure illustrated in FIG. 11 includes alayer 560, the layer 561, and the layer 563. Although FIG. 11illustrates the transistor 402 and the transistor 403 as componentsprovided in the layer 563, other components such as the transistor 404to the transistor 406 can also be provided in the layer 563.

A silicon substrate 632, an insulating layer 633, an insulating layer634, an insulating layer 635, and an insulating layer 637 are providedin the layer 563. Moreover, a conductive layer 636 is provided.

The insulating layer 634, the insulating layer 635, and the insulatinglayer 637 have functions of interlayer insulating films andplanarization films. The insulating layer 633 has a function of aprotective film. The conductive layer 636 is electrically connected tothe wiring 414 illustrated in FIG. 8 .

As the interlayer insulating film and the planarization film, forexample, an inorganic insulating film such as a silicon oxide film or anorganic insulating film of an acrylic resin, a polyimide resin, or thelike can be used. As the protective film, for example, a silicon nitridefilm, a silicon oxide film, an aluminum oxide film, or the like can beused.

For a conductive layer, a metal element selected from aluminum,chromium, copper, silver, gold, platinum, tantalum, nickel, titanium,molybdenum, tungsten, hafnium, vanadium, niobium, manganese, magnesium,zirconium, beryllium, indium, ruthenium, iridium, strontium, lanthanum,and the like; an alloy containing the above metal element; an alloycontaining a combination of the above metal elements; or the like isselected as appropriate and used. The conductor is not limited to asingle layer, and may be a plurality of layers including differentmaterials.

The Si transistors illustrated in FIG. 11 are FIN transistors eachincluding a channel formation region in the silicon substrate. FIG. 12Aillustrates a cross section in a channel width direction (a crosssection along A1-A2 illustrated in the layer 563 in FIG. 11 ). Note thateach of the Si transistors may be a planar transistor as illustrated inFIG. 12B.

Alternatively, as illustrated in FIG. 12C, a transistor including asemiconductor layer 545 of a silicon thin film may be used. Thesemiconductor layer 545 can be single crystal silicon (SOI (Silicon onInsulator)) formed on an insulating layer 546 on the silicon substrate632, for example.

The photoelectric conversion device 401 is provided in the layer 561.The photoelectric conversion device 401 can be formed over the layer563. FIG. 11 illustrates a structure where the photoelectric conversiondevice 401 uses the organic optical conductive film illustrated in FIG.10D as the photoelectric conversion layer. Note that here, the layer 567a is a cathode, and the layer 567 e is an anode.

An insulating layer 651, an insulating layer 652, an insulating layer653, an insulating layer 654, and a conductive layer 655 are provided inthe layer 561.

The insulating layer 651, the insulating layer 653, and the insulatinglayer 654 have functions of interlayer insulating films andplanarization films. In addition, the insulating layer 654 is providedto cover an end portion of the photoelectric conversion device 401, andalso has a function of preventing short circuit between the layer 567 eand the layer 567 a. The insulating layer 652 has a function of anelement isolation layer. An organic insulating film or the like ispreferably used as the element isolation layer.

The layer 567 a corresponding to the cathode of the photoelectricconversion device 401 is electrically connected to the one of the sourceand the drain of the transistor 402 included in the layer 563. The layer567 e corresponding to the anode of the photoelectric conversion device401 is electrically connected to the conductive layer 636 provided inthe layer 563 through the conductive layer 655.

The layer 560 is formed over the layer 561. The layer 560 includes alight-blocking layer 671 and a microlens array 673.

The light-blocking layer 671 can suppress entry of light into anadjacent pixel. As the light-blocking layer 671, a metal layer ofaluminum, tungsten, or the like can be used. In addition, the metallayer and a dielectric film having a function of an anti-reflection filmmay be stacked.

The microlens array 673 is provided over the photoelectric conversiondevice 401. The photoelectric conversion device 401 directly under thelens is irradiated with light passing through an individual lens of themicrolens array 673. When the microlens array 673 is provided, collectedlight can be incident on the photoelectric conversion device 401; thus,photoelectric conversion can be efficiently performed. The microlensarray 673 is preferably formed using a resin, glass, or the like havinga high light transmitting property with respect to light with awavelength subjected to imaging.

FIG. 13 is a modification example of the stack structure illustrated inFIG. 11 . FIG. 13 differs from FIG. 11 in the structure of thephotoelectric conversion device 401 included in the layer 561 and partof the structure of the layer 563. In the structure illustrated in FIG.13 , there is a bonding surface between the layer 561 and the layer 563.

The layer 561 includes the photoelectric conversion device 401, aninsulating layer 661, an insulating layer 662, an insulating layer 664,an insulating layer 665, a conductive layer 685, and a conductive layer686.

The photoelectric conversion device 401 is a pn junction photodiodeformed on a silicon substrate and includes the layer 565 b correspondingto a p-type region and the layer 565 a corresponding to an n-typeregion. The photoelectric conversion device 401 is an embeddedphotodiode, which can suppress dark current and reduce noise with thethin p-type region (part of the layer 565 b) provided on a surface side(current extraction side) of the layer 565 a.

The insulating layer 661, the conductive layer 685, and the conductivelayer 686 have functions of bonding layers. The insulating layer 662 hasfunctions of an interlayer insulating film and a planarization film. Theinsulating layer 664 has a function of an element isolation layer. Theinsulating layer 665 has a function of suppressing carrier leakage.

The silicon substrate is provided with a groove that separates pixels,and the insulating layer 665 is provided on a top surface of the siliconsubstrate and in the groove. Providing the insulating layer 665 cansuppress leakage of carriers generated in the photoelectric conversiondevice 401 to an adjacent pixel. In addition, the insulating layer 665also has a function of suppressing entry of stray light. Therefore,color mixing can be suppressed with the insulating layer 665. Note thatan anti-reflection film may be provided between the top surface of thesilicon substrate and the insulating layer 665.

The element isolation layer can be formed by a LOCOS (LOCal Oxidation ofSilicon) method. Alternatively, an STI (Shallow Trench Isolation) methodor the like may be used to form the element isolation layer. As theinsulating layer 665, for example, an inorganic insulating film ofsilicon oxide, silicon nitride, or the like or an organic insulatingfilm of polyimide resin, acrylic resin, or the like can be used. Notethat the insulating layer 665 may have a multilayer structure. Note thata structure without the element isolation layer may also be employed.

The layer 565 a (corresponding to the n-type region and the cathode) ofthe photoelectric conversion device 401 is electrically connected to theconductive layer 685. The layer 565 b (corresponding to the p-typeregion and the anode) is electrically connected to the conductive layer686. The conductive layer 685 and the conductive layer 686 each includea region embedded in the insulating layer 661. Furthermore, surfaces ofthe insulating layer 661, the conductive layer 685, and the conductivelayer 686 are planarized to be level with each other.

In the layer 563, the insulating layer 638 is formed over the insulatinglayer 637. In addition, a conductive layer 683 electrically connected tothe one of the source and the drain of the transistor 402 and aconductive layer 684 electrically connected to the conductive layer 636are formed.

The insulating layer 638, the conductive layer 683, and the conductivelayer 684 have functions of bonding layers. The conductive layer 683 andthe conductive layer 684 each include a region embedded in theinsulating layer 638. Furthermore, surfaces of the insulating layer 638,the conductive layer 683, and the conductive layer 684 are planarized tobe level with each other.

Here, main components of the conductive layer 683 and the conductivelayer 685 are preferably formed using the same metal element, and maincomponents of the conductive layer 684 and the conductive layer 686 arepreferably formed using the same metal element. In addition, maincomponents of the insulating layer 638 and the insulating layer 661 arepreferably the same.

For the conductive layer 683 to the conductive layer 686, Cu, Al, Sn,Zn, W, Ag, Pt, Au, or the like can be used, for example. In particular,Cu, Al, W, or Au is used for easy bonding. In addition, for theinsulating layer 638 and the insulating layer 661, silicon oxide,silicon oxynitride, silicon nitride oxide, silicon nitride, titaniumnitride, or the like can be used.

That is, the same metal material described above is preferably used forthe conductive layer 683 to the conductive layer 686. Furthermore, thesame insulating material described above is preferably used for theinsulating layer 638 and the insulating layer 661. With this structure,bonding where a boundary between the layer 563 and the layer 561 is abonding position can be performed.

Note that the conductive layer 683 to the conductive layer 686 may eachhave a multilayer structure of a plurality of layers; in that case,outer layers (bonding surfaces) are formed of the same metal material.Furthermore, the insulating layer 638 and the insulating layer 661 mayeach have a multilayer structure of a plurality of layers; in that case,outer layers (bonding surfaces) are formed of the same insulatingmaterial.

With this bonding, the conductive layer 683 and the conductive layer 685can be electrically connected to each other, and the conductive layer684 and the conductive layer 686 can be electrically connected to eachother. Moreover, connection between the insulating layer 661 and theinsulating layer 638 with mechanical strength can be obtained.

For bonding metal layers to each other, a surface activated bondingmethod in which an oxide film, a layer adsorbing impurities, and thelike on a surface are removed by sputtering processing or the like andcleaned and activated surfaces are brought into contact to be bonded toeach other can be used. Alternatively, a diffusion bonding method inwhich surfaces are bonded to each other by using temperature andpressure together, or the like can be used. Both methods cause bondingat an atomic level, and therefore not only electrically but alsomechanically excellent bonding can be obtained.

Furthermore, for bonding insulating layers to each other, a hydrophilicbonding method or the like can be used; in the method, after highplanarity is obtained by polishing or the like, surfaces subject tohydrophilic treatment with oxygen plasma or the like are arranged incontact with and bonded to each other temporarily, and then dehydratedby heat treatment to perform final bonding. The hydrophilic bondingmethod also causes bonding at an atomic level; thus, mechanicallyexcellent bonding can be obtained.

When the layer 563 and the layer 561 are bonded to each other, the metallayers and the insulating layers coexist on their bonding surfaces;therefore, the surface activated bonding method and the hydrophilicbonding method are performed in combination, for example.

For example, a method or the like can be used in which the surfaces aremade clean after polishing, the surfaces of the metal layers aresubjected to anti-oxidation treatment and then hydrophilicity treatment,and bonding is performed. Furthermore, hydrophilic treatment may beperformed on the surfaces of the metal layers being hardly oxidizablemetal such as Au. Note that a bonding method other than the abovemethods may be used.

The bonding allows the components included in the layer 563 to beelectrically connected to the components included in the layer 561.

FIG. 14 is a modification example of the stack structure illustrated inFIG. 13 . FIG. 14 differs from FIG. 13 in some of the structures of thelayer 561 and the layer 563.

This modification example has a structure in which the transistor 402included in the pixel 400 is provided in the layer 561. The transistor402 that is covered with an insulating layer 663 is formed using a Sitransistor in the layer 561. The one of the source and the drain of thetransistor 402 is directly connected to the one electrode of thephotoelectric conversion device 401. In addition, the other of thesource and the drain of the transistor 402 is electrically connected tothe node FD.

In an imaging device illustrated in FIG. 14 , among transistors includedin the imaging device, transistors excluding at least the transistor 402are provided in the layer 563. Although FIG. 14 illustrates thetransistor 404 and the transistor 405 as components provided in thelayer 563, other components such as the transistor 403 and thetransistor 406 can also be provided in the layer 563. Furthermore, inthe layer 563 of the imaging device illustrated in FIG. 14 , aninsulating layer 647 is provided between the insulating layer 635 andthe insulating layer 637. The insulating layer 647 has functions of aninterlayer insulating film and a planarization film.

Embodiment 7

In this embodiment, a package where an imaging portion, what is calledan image sensor chip, is put is described below.

FIG. 15A1 is an external perspective view of the top surface side of apackage in which an image sensor chip is placed. The package includes apackage substrate 410 to which an image sensor chip 452 (see FIG. 15A3)is fixed, a cover glass 420, an adhesive 430 for bonding them, and thelike.

FIG. 15A2 is an external perspective view of the bottom surface side ofthe package. A BGA (Ball grid array) in which solder balls are used asbumps 440 on the bottom surface of the package is employed. Note that,without being limited to the BGA, an LGA (Land grid array), a PGA (PinGrid Array), or the like may be included.

FIG. 15A3 is a perspective view of the package, in which parts of thecover glass 420 and the adhesive 430 are not illustrated. Electrode pads460 are formed over the package substrate 410, and the electrode pads460 and the bumps 440 are electrically connected via through-holes. Theelectrode pads 460 are electrically connected to the image sensor chip452 through wires 470.

In addition, FIG. 15B1 is an external perspective view of the topsurface side of a camera module in which an image sensor chip is placedin a package with a built-in lens. The camera module includes a packagesubstrate 431 to which an image sensor chip 451 (FIG. 15B3 is fixed, alens cover 432, a lens 435, and the like. Furthermore, an IC chip 490(FIG. 15B3 having functions of a driver circuit, a signal conversioncircuit, and the like of the imaging device is provided between thepackage substrate 431 and the image sensor chip 451; thus, a structureas an SiP (System in package) is included.

FIG. 15B2 is an external perspective view of the bottom surface side ofthe camera module. A QFN (Quad flat no-lead package) structure in whichlands 441 for mounting are provided on the bottom surface and sidesurfaces of the package substrate 431 is included. Note that thisstructure is an example, and a QFP (Quad flat package) or the above BGAmay be provided.

FIG. 15B3 is a perspective view of the module, in which parts of thelens cover 432 and the lens 435 are not illustrated. The lands 441 areelectrically connected to electrode pads 461, and the electrode pads 461are electrically connected to the image sensor chip 451 or the IC chip490 through wires 471.

The image sensor chip placed in a package having the above form can beeasily mounted on a printed board or the like, and the image sensor chipcan be incorporated in a variety of semiconductor devices and electronicdevices.

This embodiment can be combined with the description of the otherembodiments as appropriate.

Embodiment 8

With the use of a driving assistance system using the embodimentdescribed above, a driving assistance device that is suitable for asemi-autonomous driving vehicle is provided.

In Japan, the automation level of a driving assistance system forvehicles such as motor vehicles is defined in four levels, from Level 1to Level 4. Level 1 allows automation of any of acceleration, steering,and braking and is called a driving safety support system. Level 2allows automation of a plurality of operations among acceleration,steering, and braking at the same time and is called a semi-autonomousdriving system (also referred to as semi-autonomous driving). Level 3allows automation of all of acceleration, steering, and braking, where adriver handles driving only in case of emergency, and is also called asemi-autonomous driving system (also referred to as semi-autonomousdriving). Level 4 allows automation of all of acceleration, steering,and braking and is called fully autonomous driving where a driver israrely in charge of driving.

In this specification, a novel structure or a novel driving assistancesystem mainly premised on semi-autonomous driving in Level 2 or Level 3is proposed.

In order to display warnings notifying a driver of the danger inaccordance with circumstances obtained from a variety of cameras orsensors, the area of a display region adequate for the number of camerasor the number of sensors is necessary.

In addition, FIG. 16A illustrates an exterior view of a vehicle 120.Note that FIG. 16A also illustrates examples of positions where a frontimage sensor 114 a and a left-side image sensor 114L are provided.Furthermore, FIG. 16B is a schematic diagram illustrating the field offront view of a driver seen from the inside of a vehicle. A windshield110 is positioned in an upper part of the field of view of the driver,and a display device 111 having a display screen is provided in a lowerpart of the field of view.

The windshield 110 is in the upper part of the field of view of thedriver, and the windshield 110 is sandwiched between pillars 112.Although an example where the front image sensor 114 a is provided in aposition close to the field of view of the driver is illustrated in FIG.16A, without particular limitation, the front image sensor 114 a may beprovided on a front grille or a front bumper. Furthermore, although anexample of a right-hand-drive vehicle is illustrated in this embodiment,there is no particular limitation. In the case of a left-hand-drivevehicle, the front image sensor 114 a may be provided in accordance withthe position of the driver.

The image sensor chip described in Embodiment 7 is preferably used as atleast one of these image sensors.

The driver mainly looks at the display device 111 to performacceleration, steering, and braking and checks the outside of thevehicle from the windshield as an aid. As the display device 111, anyone of a liquid crystal display device, an EL (Electro Luminescence)display device, and a micro LED (Light Emitting Diode) display device isused. Here, an LED chip whose one side size is larger than 1 mm iscalled a macro LED, an LED chip whose one side size is larger than 100 mand smaller than or equal to 1 mm is called a mini LED, and an LED chipwhose one side size is smaller than or equal to 100 m is called a microLED. It is particularly preferable to use a micro LED as an LED elementapplied to a pixel. The use of a micro LED can achieve an extremelyhigh-resolution display device. The display device 111 preferably hashigher resolution. The pixel density of the display device 111 can be apixel density of higher than or equal to 100 ppi and lower than or equalto 5000 ppi, preferably higher than or equal to 200 ppi and lower thanor equal to 2000 ppi.

For example, a center part 111 a of the display screen of the displaydevice displays an image obtained from an imaging device provided at thefront outside the vehicle. In addition, parts 111 b and 111 c of thedisplay screen perform meter display such as display of speed, estimateddistance to empty, and abnormality warning. Furthermore, video of theleft side outside the vehicle is displayed in a lower left part 111L ofthe display screen, and video of the right side outside the vehicle isdisplayed on a lower right part 111R of the display screen.

The lower left part 111L of the display screen and the lower right part111R of the display screen can eliminate side view mirror protrusionsthat protrude greatly outside the vehicle by computerization of sideview mirrors (also referred to as door mirrors).

The display screen of the display device 111 may be configured to beoperated by touch input so that part of video is enlarged or reduced, adisplay position is changed, or the area of the display region isexpanded, for example.

Because an image on the display screen of the display device 111 is acomposite of data from a plurality of imaging devices or sensors, theimage is created with an image signal processing device such as a GPU.

With the use of the driving assistance system described in Embodiment 1,an enlarged highlighted image can be output to the display device 111 byacquisition of monochrome image data with a wide dynamic range,extraction of only a distant region, and colorization performed byinference.

By using AI as appropriate, the driver can mainly look at a displayedimage on the display device, that is, an image utilizing the imagesensors and the AI in operating the vehicle and look at the front of thewindshield as an aid. Operating the vehicle while looking at imagesutilizing the AI, rather than driving with only the driver's eyes, canbe safe driving. Moreover, the driver can operate the vehicle with asense of security.

Note that the display device 111 can be used around a driver's seat(also referred to as a cockpit portion) in various types of vehiclessuch as a large-sized vehicle, a middle-sized vehicle, and a small-sizedvehicle. Furthermore, the display device 111 can also be used around thedriver's seat in a vehicle such as an airplane or a ship.

In addition, although this embodiment describes an example where thefront image sensor 114 a is placed below the windshield, there is noparticular limitation. An imaging camera illustrated in FIG. 17 may beplaced on a hood or around an in-vehicle rearview mirror.

The imaging camera in FIG. 17 can also be referred to as a dashboardcamera, which includes a housing 961, a lens 962, a support portion 963,and the like. When a double-sided tape or the like is attached to thesupport portion 963, the imaging camera can be placed in the windshield,the hood, a rearview mirror support, or the like.

When the imaging camera in FIG. 17 is provided with the image sensor,driving video can be stored in the inside of the imaging camera or anin-vehicle storage device.

This embodiment can be freely combined with the other embodiments.

REFERENCE NUMERALS

-   10: data acquisition device, 11: solid-state imaging element, 12:    analog arithmetic circuit, 13: A/D circuit, 14: memory portion, 15:    display device, 16 a: neural network portion, 16 b: neural network    portion, 16 c: neural network portion, 17: data extraction portion,    18 a: storage portion, 18 b: storage portion, 18 c: storage portion,    19: display portion, 21: display example, 22: distant region, 23:    colorization image, 24: highlighted image, 31: driving assistance    system, 41: imaging system, 110: windshield, 111: display device, 11    a: central part, 111 b: part, 111 c: part, 111L: lower left part,    111R: lower right part, 112: pillar, 114 a: front image sensor,    114L: left-side image sensor, 120: vehicle, 200: pixel block, 201:    circuit, 202: capacitor, 203: transistor, 204: transistor, 205:    transistor, 206: transistor, 207: resistor, 211: wiring, 212:    wiring, 213: wiring, 215: wiring, 216: wiring, 217: wiring, 218:    wiring, 219: wiring, 300: pixel array, 301: circuit, 302: circuit,    303: circuit, 304: circuit, 305: circuit, 306: circuit, 311: wiring,    400: pixel, 401: photoelectric conversion device, 402: transistor,    403: transistor, 404: transistor, 405: transistor, 406: transistor,    407: capacitor, 410: package substrate, 411: wiring, 412: wiring,    413: wiring, 413 a: wiring, 413 b: wiring, 413 c: wiring, 413 d:    wiring, 413 e: wiring, 413 f: wiring, 413 g: wiring, 413 h: wiring,    413 i: wiring, 413 j: wiring, 414: wiring, 415: wiring, 417: wiring,    420: cover glass, 421: wiring, 422: wiring, 423: wiring, 424:    wiring, 430: adhesive, 431: package substrate, 432: lens cover, 435:    lens, 440: bump, 441: land, 450: transistor, 450 a: transistor, 450    b: transistor, 450 c: transistor, 450 d: transistor, 450 e:    transistor, 450 f: transistor, 450 g: transistor, 450 h: transistor,    450 i: transistor, 450 j: transistor, 451: image sensor chip, 452:    image sensor chip, 460: electrode pad, 461: electrode pad, 470:    wire, 471: wire, 490: IC chip, 545: semiconductor layer, 546:    insulating layer, 560: layer, 561: layer, 563: layer, 565 a: layer,    565 b: layer, 566 a: layer, 566 b: layer, 566 c: layer, 566 d:    layer, 567 a: layer, 567 b: layer, 567 c: layer, 567 d: layer, 567    e: layer, 632: silicon substrate, 633: insulating layer, 634:    insulating layer, 635: insulating layer, 636: conductive layer, 637:    insulating layer, 638: insulating layer, 647: insulating layer, 651:    insulating layer, 652: insulating layer, 653: insulating layer, 654:    insulating layer, 655: conductive layer, 661: insulating layer, 662:    insulating layer, 664: insulating layer, 665: insulating layer, 671:    light-blocking layer, 673: micro lens array, 683: conductive layer,    684: conductive layer, 685: conductive layer, 686: conductive layer,    961: housing, 962: lens, and 963: support portion.

1. A driving assistance system comprising: a step of driving a vehicleincorporating an imaging device; a step of capturing a monochrome imageof a front of the vehicle during driving by the imaging device; a stepof performing inference of a region of at least a sky, a car, and a roadby performing segmentation processing on a monochrome image includingthe distant region; a step of performing inference of a specific distantregion by performing depth estimation processing on the monochrome imageincluding the distant region; a step of determining a center of aportion cut off from the monochrome image on the basis of thesegmentation processing and the depth estimation processing; a step ofextracting a rectangular region in which a center is a central portion,inputting extracted data, and performing super-resolution processing; astep of inputting an output result of the super-resolution processingand performing colorization processing for highlighting of an objectincluded in the distant region with high accuracy; and a step ofperforming enlargement of the distant region subjected to colorizationprocessing.
 2. The driving assistance system according to claim 1,wherein the segmentation processing uses first neural networkprocessing, wherein the depth estimation processing uses second neuralnetwork processing, wherein the super-resolution processing uses thirdneural network processing, and wherein the colorization processing usesfourth neural network processing.
 3. The driving assistance systemaccording to claim 1, wherein the imaging device does not include acolor filter.
 4. The driving assistance system according to claim 1,further comprising a step of measuring driving speed of the vehicle. 5.The driving assistance system according to claim 1, wherein size of therectangular region in which the center is the central portion isdetermined by the driving speed of the vehicle.
 6. The drivingassistance system according to claim 1, wherein the specific distantregion includes at least a road edge portion.
 7. A program in which acomputer executes each step, according to the driving assistance systemof claim 1.